The graphic images used in computer-generated displays have been stored in image memories at address locations mapping respective points at regular intervals along the raster scanning of a display image space. Each addressed location in image memory has contained a digital word, at least a portion of which has encoded the brightness, hue and saturation of a color picture element (or "pixel") at the corresponding point in image space (and, in run-length encoding schemes, the value of succeeding pixels). A number of different schemes for encoding the brightness, hue and saturation of color pixels exist in the prior art.
One may analyze each color pixel as the sum of the three additive primary colors, red, green and blue, for example. The amplitudes of the red (R), green (G) and blue components may each be coded in a number n of bits, n normally being in the range five to eight inclusive. Coding may be linear, logarithmic, or in accordance with some other function. It is also known to linearly encode red, green and blue in different numbers p, q and r of bits depending on their relative contributions to luminance. Encoding green in seven bits, red in six bits and blue in five bits is an example of such coding, which provides luminance with nearly eight bits of amplitude resolution. There are 217 levels in luminance assuming it to be comprised of 59% green, 30% red and 11% blue in accordance with the NTSC standard. This compares favorably with the 219 levels in luminance for eight-bit coding of luminance using level 35 as black and level 254 as white, as recommended by the Society of Motion Picture and Television Engineers (SMPTE). This amplitude resolution in luminance (Y=0.29R+0.59G+0.11B) is generally considered to make quantization noise imperceptible to the human viewer. The reader is referred to M. F. Cowlishaw's paper "Fundamental Requirements of Picture Presentation" appearing on pages 101-107 of PROCEEDINGS OF THE SID, Vol. 26/2, 1985, for a comprehensive treatment of coding additive primary colors in differing numbers of bits.
One may analyze each color pixel as the sum of a luminance-only primary color and two chrominance-only primary colors. The luminance-only primary represents whiteness or brightness of the pixel. The chrominance-only primaries do not correspond with any real color, but together are representative of the difference of any real color from the luminance-only primary. Analyzing pixels in terms of luminance and two color-difference signals is usually done so that one may encode the two color-difference signals in less bandwidth than luminance. In digital sampled-data terms this usually corresponds to chrominance being sampled in image space at a subsampled rate compared to luminance being sampled in image space. Subsampling may be in the direction of line advance as well as in the direction in which scan lines extend.
A small computer may be used to decode television display material that has been encoded in an economical format (e.g., to permit the transmission of image data via telephone lines or the recording of image data on compact disc). Such image data is referred to as "compressed video". This small computer may be provided with general-purpose memory, portions of which are available for use as image memory to provide buffering between an irregular flow of received image data and the regular flow of image data to the display. It is desirable to provide an image memory configuration that is well suited to being used interchangeably with other data storage in general-purpose memory and does not require the use of dedicated portions of the memory for image storage.
The encoding of television information for transmission over media of such limited bandwidths as those available from a telephone line or compact disc forces the designer to resort to powerful video compression methods. These methods rely upon transmitting as little new image information per frame as possible and upon storing as much old image information as possible; and transmission of new image information cannot be done, at least not entirely, in real time. In order to write a display in real time, then, it is essential then to have frame buffer storage memory with the capability of storing at least two frames of video information. Such memory can be written to from a flow of compressed image data received in non-real time and read from so as to supply the display apparatus with a regular flow of image data in real time. The frames of stored information are updated on one frame and used to support display on the next, in staggered phasing. The frame buffer storage memory is bit-map-organized for convenience in constructing updated images from previous image data in accordance with instructions included in the compressed video data.
Image memories, the addressable storage locations of which map corresponding picture elements or "pixels" on a display screen and which store single bits descriptive of whether those corresponding pixels are bright or dark, have been described as being "bit-map-organized" for many years. In recent years the term "bit-map-organized" has been applied to certain image memories in which a pixel variable related to brightness is not expressed in terms of a single bit, but rather in terms of a plurality of bits. Such brightness-related variables may be luminance variables or may be color-difference variables used in connection with describing color displays, for example; or they may be component color variables, as another example. The term "bit-map-organized" has been extended to refer to two different memory configurations, each storing a plural-bit value descriptive of a pixel variable.
A plural-bit-variable bit-map-organized image memory of a first general type known in the prior art can be thought of as employing a number of planes, which number equals the total number of bits in the plural-bit-variable(s) describing a single pixel. The most significant bits of a first of the pixel variables are stored in the first bit plane at storage locations having respective addresses mapping respective pixel locations in the display; the next most significant bits of the first pixel variable are stored in the second bit plane at storage locations having respective addresses mapping respective pixel locations in the display in a manner corresponding to the mapping of the storage locations in the first bit plane; and so forth, proceeding to less significant bits in the first pixel variable, then proceeding through the bits of each other pixel variable (if any) proceeding from most significant to least significant bit. Responsive to a single address this type of memory furnishes simultaneously the respective plural bits of all the pixel variables descriptive of a particular pixel. Essentially, the spatial positions of individual pixels in the display have a one-to-one correspondence with respective image memory addresses, in a spatial mapping. This spatial mapping is held together by the tracing of the display screen and scanning of image memory addresses each being done in accordance with a prescribed pattern of correspondence betweenthese activities. As long as the pattern of correspondence between these activities is adhered to, the rate at which and order in which these activities are carried out do not affect the spatial mapping between the image memory addresses and the spatial positions of display pixels.
A second general type of plural-bit-variable bit-map-organized image memory known to the prior art does not require a one-to-one correspondence between image memory address and the spatial positions of display pixels. There is a list of the values of the plural-bit pixel variables in a prescribed cyclic order, which cycles are arranged in the sequence of the tracing of the spatial positions of pixels in the display. The list is converted to a string of values of the pixel variables, with the bits in each value arranged in prescribed order according to relative significance. Each string of values is divided into words of given bit length, which words are stored respectively in successively addressed locations in the image memory. An image memory of this second general type has to be read out to a formatter with pixel unwrapping capability. The formatter reconstitutes the words into a string of values which are then parsed back into successive values of each pixel variable. The variables for each pixel are temporally aligned by the formatter to be available at the time the spatial position of that pixel is reached in the scanning of the display screen.
When a pixel is described in terms of plural variables--e.g., three primary color variables, or a luminance variable and two chrominance variables--it has been a general practice to group these variables in a prescribed order for each pixel and to use each group as subvariable components of a respective value of a complex pixel-descriptive variable. The values of this complex variable are then stored in a bit-map-organized image memory organized as either the first or the second type of image memory described above. This practice is reasonably satisfactory as long as the pixel-descriptive variables used as subvariable components of the complex variable are sampled at corresponding points in display space and with the same sampling density. However, it is desirable to be able to sample the pixel variables at differing sampling densities. This is primarily so that compressed video data can be more efficiently coded; bit image memory is also conserved and image processing involves less calculation. If the pixel variables have different sampling densities in image space, using complex pixel-descriptive variables becomes unattractive.
A type of dual-ported, dynamic random-access memory that has recently become commercially available is the so called "video random-access memory" or "VRAM". This dynamic memory, in addition to a random-access input/output port through which information can be written into or read out of the memory, has a serial-access port from which a row of data can be read serially at video scan rates. The row busses of a principal dynamic random-access memory portion of a VRAM are arranged to transfer data in parallel to a smaller auxiliary memory of the VRAM, during an interval equal to the read interval from the random-access port. A counter is provided in each VRAM for scanning the addresses of the auxiliary memory during its reading, so the auxiliary memory can function as a shift register. After parallel loading of the auxiliary memory, its contents are read out serially through the VRAM serial output port, with the counter counting at a relatively high clock rate. This clock rate can be the rate at which the luminance-only picture elements are delivered to the display monitor of the computer apparatus, for example. This speed of reading is possible because the capacitance-to-substrate of the auxiliary memory busses is relatively low owing to the smaller size of this auxiliary memory.
In some types of VRAM, data can also be serially read into the auxiliary memory via the serial-access port, to be transferred in parallel into the principal dynamic random-access portion of the VRAM. This allows faster writing of the VRAM than is possible by writing information via its random-access port.
D. L. Sprague, N. J. Fedele and L. D. Ryan in their U.S. patent application Ser. No. 4,740,832 entitled "IMAGE STORAGE USING SEPARATELY SCANNED LUMINANCE AND CHROMINANCE VARIABLES", filed Oct. 14, 1986, and assigned to RCA Corporation, point out that time interleaving of lines of digitized chrominance information with lines of digitized luminance information is usefully applied to the reading of VRAM through its serial access port. Such practice facilitates the use of separate bit-map organizations of the second general type for luminance and chrominance variables in VRAM. The use of separate bit-map organizations for luminance and chrominance variables, Sprague et al. perceived, avoids the problems encountered in the use of complex pixel-descriptive variables in a unified bit-map organization when chrominance is sampled less densely in display space than luminance is. Sprague et al. describe in detail how to linearly pack image memory into VRAM, when separate bit-map organizations for luminance and chrominance variables are used. The use of separate bit-map organizations can be accommodated by a display processor following the VRAM and responding to the image data supplied from the VRAM serial port, which display processor uses a rate-buffering memory for at least the chrominance samples, Sprague et al. realized. They used the rate-buffering memory for a further purpose to provide the sample bed information to support spatial interpolation in the direction transverse to line scan.
The literature indicates that 32 levels of luminance detail resolution in linearly coded video is all that is needed to support displays that the human viewer cannot distinguish from broadcast-quality television displays. The reader is directed to the paper by D. E. Troxel et al. entitled "A TWO-CHANNEL PICTURE CODING SYSTEM: I--REAL TIME IMPLEMENTATION", which appeared on pages 1841-1848 of IEEE TRANSACTIONS ON COMMUNICATIONS, Vol. COM-24, No. 12, December 1981. They indicate five bits per sample can linearly code luminance highs so pictures reconstructed from digitized samples are indistinguishable from broadcast-quality originals irrespective of viewing distance. If viewing distance is at least four times picture height, four bits can suffice.
Rather than coding a full-bandwidth luminance signals in eight-bits and two narrowband chrominance signals in substantially the same number of bits, in order to obtain a high-quality display, one may alternatively code a full-bandwidth luminance detail signal in four to five bits, a narrowband luminance signal in eight bits and the two narrowband chrominance signals in eight bits, the present inventors suggest. In such case, the VRAM will store luminance detail data in a bit-map organization that fully samples image space, and the VRAM will store narrowband luminance and chrominance data in at least one bit-map organization that subsamples image space. The Sprague et al. display processor will be modified to include rate-buffering memory and interpolator circuitry for narrowband luminance data as well as for narrowband chrominance data.
Suppose, for example, the display has 241 active scan lines of 253 pixels each and chrominance signals are subsampled 4:1 in the direction in which scan lines extend and in the direction transverse to scan lines. If the Sprague et al. scheme of coding images were followed with eight-bits alloted to luminance and to the two chrominance signals, there would be 241.times.253.times.8=487,784 bits of luminance information per frame and 61.times.63.times.8=30,744 bits of information per frame for each chrominance component. This is a total of 549,272 bits per frame when this form of the Sprague et al. coding scheme is employed. Suppose one follows the suggestion of the present inventors, coding narrowband luminance in eight-bits and luminance detail in five-bits. The number of bits of narrowband luminance is 61.times.63.times.8=30,744 bits, just as for each chrominance component. The number of bits of luminance detail is 241.times.253.times.5=304,865 bits.
This results in a total of 397,099 bits per frame, which is almost a one-quarter reduction in the number of bits per frame over the coding scheme used by Sprague et al. This surprisingly high reduction in number of bits per frame comes about because the narrowband luminance is narrowband not only in the direction in which scan lines extend, but also in the direction transverse to scan lines. This reduction in number of bits per frame can be viewed as being an increase in number of frames that can be described by a long bit stream.
A corresponding increase in number of frames that can be described by a long bit stream will be noted in the compressed video. This is the primary reason for seeking efficiency in the coding of image data stored in image memory. Usually all calculations in the drawing processor take place at a rate determined by the time needed to process the pixel variable with most bits of amplitude resolution. If calculations are made concerning a pixel variable with fewer bits amplitude resolution, it is customary to pad the variable with added ZEROs. This is the easiest procedure when image calculations are carried out separately and serially on component portions of the image data. When image calculations are carried out separately and parallelly on component portions of the image data, padding variables to equal bit length is still the easiest procedure, since the calculating hardware may be simply replicated and then operated synchronously. Efficient image coding can allow denser packing of image memory, but the formatter following image memory must have a more sophisticated pixel unwrapping capability if dissimilar bit lengths are used in the various pixel variables. Especially as the cost of computer memory per bit falls with progress in the art of making random-access memory, it may be preferred to pad the pixel variables stored in image memory to even bit-lengths, where their bit lengths differ by only one or two bits, in order to simplify timing and formatting problems.
One must take into account that the drawing processor updating VRAM contents responsive to compressed video has to perform a greater number of calculations to process four pixel variables than is required to process three pixel variables. However, the restrictions upon the transmission medium for compressed video are usually more severe system constraints than providing for a faster and somewhat more complex drawing processor. The four variable pixel descriptions advocated by the present inventors provide a better accommodation to the condition where the image changes cannot be completely accommodated by the compressed video medium. One can code the images in terms of the three narrowband variables and dispense with the wideband luminance variable for a time. A succession of images may lack detail, but can change fast enough that the displayed image is not segmented into different time frames. By way of illustration, continue the previously discussed example of image coding in accordance with the present inventors' precepts. The 30,744 bits of information per frame for each of the narrowband pixel variables sum to 92,232 bits per frame, somewhat less than one-quarter the number of bits per frame needed for the full-bandwidth image.
If one codes pixels in terms of a luminance detail variable and three narrowband color-componen variables as the present inventors suggest above, these three narrowband variables need not be a luminance variable and two color-difference variables. Indeed there are reasons they should not be.
Coding pixel information in terms of luminance and chrominance is inefficient in terms of providing optimal resolution in luminance for given pixel code lengths. Amplitude resolution is needed more in luminance components of the drive signals to the display device than in color-difference components of the drive signals insofar as keeping the human observer unaware of quantizing noise is concerned. When the display color is white or shades thereof, the chrominance variables are non-contributory to the display. The resolution in luminance is defined just by the number of available bit places in the luminance variable alone when pixel colors are defined in terms of luminance and chrominance variables.
In red (R), green (G) and blue (B) variables if red and blue have the same number of bits defining their ranges of variation as green does, the resolution in luminance amplitude is determined by the number of bit places in green, the most significant component of luminance, divided by the fraction of luminance that is green. This fraction is customarily about 0.59. Eight bits in luminance can define 256 levels therein; 7 bits, 128 levels. Eight bits in green can define 434 levels approximately in luminance; seven bits in green can define 217 levels approximately in luminance; six bits in green can define 108 levels approximately in luminance. This advantage in luminance resolution is not lost when the number of bit places in red and in blue is reduced relative to the number of bit places in green, towards the number of bit places in the primary color being in proportion to the logarithm-base-two of their respective contributions to luminance, as advocated by Cowlishaw. In fact, because the red and blue color variables introduce quantization levels into luminance that are intermediate to those introduced by the green color variable, there is somewhat better resolution in luminance. (Where the number of bits in red, green and blue are the same an improvement in luminance resolution can be obtained by staggering the quantizing levels in luminance respectively contributed by red, green and blue color variables, it is pointed out.)
One can compare the coding of luminance detail signal together with narrowband red, green and blue signals against the previously described coding of luminance detail signal together with narrowband luminance signal and two narrowband color difference signals. Assume narrowband red, green and blue are coded in six, seven and five-bits per pixel in accordance with Cowlishaw and that luminance detail is coded in five-bits per pixel in accordance with Troxel et al. Again, a display having 241 active scan lines of 253 pixels each for full bandwidth pixels is assumed. There are 241.times.253.times.5=304,865 bits as before. There are 61.times.63.times.6=23,058 bits per frame of narrowband red information, 61.times.63.times.7=26,901 bits per frame of narrowband green information, and 61.times.63.times.5=19,215 bits per frame of narrowband blue information. This is a total of 69,174 bits per frame of narrowband color component information, as compared to the 92,232 bits per frame of narrowband information for narrowband luminance and chrominance variables. This reduces the 397,099 bits per frame needed to code luminance and chrominance variables to 304,865+69,174=374,039, coming closer to the one-quarter reduction in the number of bits per frame over the coding scheme used by Sprague et al. What is more important is that when the amount of frame-to-frame change in image overruns the capacity of the compressed video channel, the cutback to narrowband transmission can reduce the number of bits per frame over seven times.